Implement Azure Data Lake Storage (ADLS) cloud connections#183
Open
Edwardvaneechoud wants to merge 4 commits intomainfrom
Open
Implement Azure Data Lake Storage (ADLS) cloud connections#183Edwardvaneechoud wants to merge 4 commits intomainfrom
Edwardvaneechoud wants to merge 4 commits intomainfrom
Conversation
This commit adds comprehensive support for ADLS Gen2 storage with multiple authentication methods, full read/write capabilities, and local testing support via Azurite emulator. Backend changes: - Add azure_sas_token field to cloud storage schemas and database models - Implement ADLS storage options for access_key, service_principal, and sas_token auth - Add ADLS path parsing utilities (az:// and abfs:// URI formats) - Add Azure Blob Storage client integration with file listing capabilities - Update cloud storage reader to support ADLS paths alongside S3 - Store and retrieve ADLS secrets (account_key, client_secret, sas_token) Worker changes: - Add ADLS storage options to worker models with secret decryption - Enable ADLS write operations for all file formats (Parquet, CSV, JSON, Delta) - Support all ADLS authentication methods in worker context Frontend changes: - Enable ADLS storage type option in cloud connection UI - Add UI fields for Azure-specific credentials (account name, keys, tenant, etc.) - Add SAS token authentication support with password field visibility toggle - Add azure_sas_token to TypeScript type definitions and API conversions - Update form validation for ADLS connection requirements Dependencies: - Add azure-storage-blob ^12.24.0 for Blob Storage SDK - Add azure-identity ^1.19.0 for service principal authentication - Add adlfs ^2025.1.0 for Polars integration Testing infrastructure: - Create test_utils/adls module for Azurite-based testing - Add start_azurite and stop_azurite CLI commands - Implement test data generator with Parquet, CSV, and JSON samples - Add comprehensive README with usage examples and troubleshooting - Create managed_azurite context manager for integration tests Supported features: - Multiple authentication methods: access key, service principal, SAS token - Read operations: single files and directory wildcards - Write operations: Parquet, CSV, JSON, Delta Lake formats - Path formats: both az:// and abfs:// URI schemes - Local testing with Azurite emulator (Docker-based) - Secure secret storage with per-user encryption Files modified: 9 backend/frontend files Files created: 5 test utility files
✅ Deploy Preview for flowfile-wasm canceled.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit adds comprehensive support for ADLS Gen2 storage with multiple
authentication methods, full read/write capabilities, and local testing support
via Azurite emulator.
Backend changes:
Worker changes:
Frontend changes:
Dependencies:
Testing infrastructure:
Supported features:
Files modified: 9 backend/frontend files
Files created: 5 test utility files